Fractional Max-Pooling

Convolutional networks almost always incorporate some form of spatial pooling, and very often it is α × α max-pooling with α = 2. Max-pooling act on the hidden layers of the network, reducing their size by an integer multiplicative factor α. The amazing by product of discarding 75% of your data is that you build into the network a degree of invariance with respect to translations and elastic distortions. However, if you simply alternate convolutional layers with max-pooling layers, performance is limited due to the rapid reduction in spatial size, and the disjoint nature of the pooling regions. We have formulated a fractional version of max-pooling where α is allowed to take non-integer values. Our version of maxpooling is stochastic as there are lots of different ways of constructing suitable pooling regions. We find that our form of fractional max-pooling reduces overfitting on a variety of datasets: for instance, we improve on the state of the art for CIFAR-100 without even using dropout. 1 CONVOLUTIONAL NEURAL NETWORKS Convolutional networks are used to solve image recognition problems. They can be built by combining two types of layers: • Layers of convolutional filters. • Some form of spatial pooling, such as max-pooling. Research focused on improving the convolutional layers has lead to a wealth of techniques such as dropout (Srivastava et al., 2014), DropConnect (Wan et al., 2013), deep networks with many small filters(Ciresan et al., 2012), large input layer filters for detecting texture (Krizhevsky et al., 2012), and deeply supervised networks (Lee et al., 2014). By comparison, the humble pooling operation has been slightly neglected. For a long time 2 × 2 max-pooling (MP2 has been the default choice for building convolutional networks. There are many reasons for the popularity of MP2-pooling: it is fast, it quickly reduces the size of the hidden layers, and it encodes a degree of invariance with respect to translations and elastic distortions. However, the disjoint nature of the pooling regions can limit generalization. Additionally, as MP2-pooling reduces the size of the hidden layers so quickly, stacks of back-to-back convolutional layers are needed to build really deep networks (Lin et al., 2014; Simonyan & Zisserman, 2014; Szegedy et al., 2014). Two methods that have been proposed to overcome this problems are: • Using 3× 3 pooling regions overlapping with stride 2 (Krizhevsky et al., 2012). • Stochastic pooling, where the act of picking the maximum value in each pooling region is replaced by a form of size-biased sampling (Zeiler & Fergus). However, both these techniques still reduce the size of the hidden layers by a factor of two. It seems natural to ask if spatial-pooling can usefully be applied in a gentler manner. If pooling was to only reduce the size of the hidden layers by a factor of √ 2, then we could use twice as many layers of pooling. Each layer of pooling is an opportunity to view the input image at a different scale. Viewing images at the ‘right’ scale should make it easier to recognize the tell-tale features that identify an object as belonging to a particular class.